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Background / Context: 

Description of prior research and its intellectual context. 



A central goal of instruction is to enable learners to transfer acquired knowledge to appropriate 
future situations. One factor that likely promotes far transfer is conceptual coherence (cf. 

Murphy & Medin, 1985). For elementary and middle-school school children in middle-high-SES 
schools, “explicit” instruction on the Control of Variables Strategy (CVS) that emphasizes 
understanding of the rationale behind CVS has proven to be effective at promoting transfer to 
different contexts, even after extended time delays (e.g., Strand-Cary & Klahr, 2008; Klahr & 
Nigam, 2004; Chen & Klahr, 1999). An understanding of this rationale may increase conceptual 
coherence by providing linkage between the procedural rules of CVS (i.e., by explaining why 
only the focal variable should be contrasted). However, when the same instruction was delivered 
to students in low-SES schools, near — but especially far — transfer rates were much lower (e.g., 
Klahr & Li, 2005). We hypothesized that the poorer performance on far transfer assessments was 
primarily due to students’ failure to develop a generalized understanding of the rationale for 
controlling variables, resulting in weakly-integrated procedural knowledge. 



Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

The purpose of the study was to see whether the addition of more focused questions in 
computerized instruction would improve far transfer performance of low-SES students. These 
additional questions prompted students to consider whether the causality of a variable could be 
determined from various experimental set-ups and thereby an understanding of the rationale for 
controlling other variables. 

In addition, we compared learning and transfer outcomes of these two versions of 
computerized instruction to what we considered to be an instructionally good control lesson. The 
focus of this control lesson was also on the rationale for controlling variables. However, the 
overall percent of time devoted to considering the rationale for controlling variables was less in 
this control lesson than in the computerized instruction. 

Setting: 

Description of the research location. 

Research was conducted at a local Pittsburgh science and technology magnet school serving 
students in grades six through nine. Students given the computerized instruction worked in the 
school’s computer lab, and the curricular Control lesson primarily took place in students’ regular 
science classroom. Each phase of the study occurred during students’ regular science class 
period. 

Research Design: 

Description of the research design. 
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The overall design was quasi-experimental, where all students in one class were given the 
regular curriculum lesson, and all students in the other class were given the computerized 
instruction (delivered by the “TED” tutor, for “Training in Experimental Design”). 

In the class given the computerized TED instruction, students were randomly assigned to 
either the baseline or “added-questions” version of the computer tutor. TED instruction consisted 
primarily of evaluating presented experiments. The added-questions condition was identical to 
the baseline version, but included five additional questions throughout the instruction focusing 
on the causal logic of CVS. 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 

Thirty-three students in two sixth-grade science classes participated and completed all parts of 
the intervention and assessments (19 completed the computerized TED instruction, and 14 
completed the Control lesson, a teacher-delivered hand-on curricular lesson, from the textbook 
“Foundations of Physical Science” published by the Cambridge Physics Outlet and approved by 
the Pittsburgh Public School District). By administrative design, the classes given the 
computerized TED and Control instruction were similar in terms of such variables as gender 
composition (32% and 36% female, respectively) and ethnicity (58% African American, 37% 
Caucasian, and 5% Asian; 50% African American, 43% Caucasian, and 7% Asian, respectively). 
The majority of students (55%) were eligible for free or reduced-price lunch; thus, this was a 
primarily low-SES population. 



Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and duration. 

For Track 2, this may include the development and validation of a measurement instrument. 

All students : The procedure is summarized in Table 1. On Day 1, all students were first 
introduced to the lesson topic by individually answering questions on the scientific method by 
referencing textbook content, and then discussed their answers as a whole-classroom activity. 
Following this, all students completed the computerized story pretest to assess initial 
understanding of experimental design. 

TED conditions instruction : After completing the story pretest, students in both 
computerized TED conditions began the computerized instructional phase, in which they first 
completed a ramps pretest in which they designed four experiments, one for each of four ramps 
variables. Afterwards they were presented with three different experimental comparisons using 
ramps. For each they responded to whether or not it was a “good way” to find out about the focal 
variable, typed an explanation for their response, then responded to whether or not it would 
allow them to “know for sure” whether the focal variable caused a hypothetical different 
outcome, and typed an explanation for this response. Then students received feedback on their 
responses. 

Afterwards, students in the added-questions condition were asked to select all of the 
variables that could have caused this outcome. They received feedback on their response. 
Students in the baseline condition heard a procedural explanation instead. 

All students then heard a full explanation for why the given set-up was or was not a good 
experiment. 
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On Day 2 of instruction, the evaluation instructional sequence was repeated for the 
second and third virtually-presented set-ups. For both the second and third set-ups, which were 
confounded, after the above instructional sequence was completed, the experimental confounds 
were fixed. Then the whole sequence was repeated for the unconfounded designs. Thus, in total, 
students in the added-questions condition responded to five additional prompts. 

Finally, students in both computerized conditions completed the ramps posttest to assess 
near-transfer performance. 

Control condition instruction : After completing the story pretest, the teacher led the class 
in reviewing the next section of the CPO textbook that included activities such as identifying the 
variables in an experiment of carts rolling down ramps, discussing “controlled experiments” (and 
why controlling variables is necessary). 

Day 2 began with a class discussion of the relationship between a ramp’s height and 
speed at which something rolls down it. The class predicted a positive relationship between these 
variables. Then students were assigned to groups and to a ramps apparatus. Each ramp was 
preset to a different height (though other variables were not controlled). Students were asked to 
time how long the cart took to roll down the ramp. Once all groups were finished, the class 
reconvened and each group stated their result. The positive relationship they initially predicted 
was contradicted by their results. This led to a discussion of the problems with the experiment, 
which focused on experimental confounds. After the class came to an agreement about how to 
fix the experimental confounds, students worked in their original groups to re-run the 
experiment. The class again reconvened to discuss their results, which were consistent with their 
initial prediction. 

All students : On Day 3, all students completed the computerized story posttest (identical 
to the story pretest) to assess their ability to apply what they learned to several other domains. 

Three weeks later, all students completed a paper far-transfer follow-up posttest, which 
required application to three novel domains and included more variables per problem than in the 
story pre- and posttests. Two standardized test items were also included in the follow-up posttest. 
This test was administered in the regular science classroom rather than in the computer lab to 
remove any possible context effect advantages for students in the TED conditions. 



Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

For Track 2, this may include the use of existing datasets. 

All students’ interface responses (on the story pretest, the ramps pretest, during the 
instructional phases, ramps posttest, and story posttest) were saved as log files, imported into 
spreadsheets, and later analyzed. Students’ responses on the follow-up posttest were scored by a 
human coder. The teacher-delivered Control lesson was video-taped. 

For the TED conditions, each unconfounded experiment testing the given focal variable 
that students designed on the ramps pre- and posttests was scored as one point (for a maximum 
score of 4). For all students, story pre and posttests were scored by assigning one point for each 
unconfounded experiment students designed (out of 3 points), one point was given for each set- 
up students converted into an unconfounded design (out of 2 points), and one point was given for 
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students correctly evaluating the unconfounded design as a good experiment, for a maximum 
score of 6. The delayed posttest was scored in a similar manner. 

Analysis of covariance (covarying for pretest scores and a measure of deductive 
reasoning ability) was conducted to determine condition effects and to test for any interactions. 



Findings / Results: 

Description of the main findings with specific details. 

TED conditions : As predicted, there was no difference between the baseline and added- 
questions versions of the computerized instruction on the measure of near-transfer performance, 
the ramps posttest. Mastery rates (i.e., designing at least 3 out of 4 unconfounded set-ups) were 
similarly high for the added-questions and baseline conditions (88% and 83%, respectively). 

All conditions : However, there was a significant main effect of condition for the 
immediate story posttest. Pairwise comparisons indicate that students in the added-questions 
condition scored significantly higher than students in the Control condition, and marginally 
higher than students in the TED baseline condition. Furthermore, students in the baseline TED 
condition scored marginally higher than students in the Control condition. 

On the delayed posttest, there was a condition by story pretest interaction (shown in 
Figure 1), where there was no relationship between story pretest and delayed posttest for students 
in the added-questions condition (r(8) = +.22, p = .57), but similar positive correlations for the 
baseline TED and Control conditions (r(9) = +.67 , p = .03; r( 1 3) = +.19, p = .001, respectively). 

This benefit of the added-questions manipulation over the other conditions was greatest 
for the lowest-knowledge students, or those students who initially showed no understanding of 
any aspects of CVS (i.e., identifying the focal variable, contrasting the focal variable and 
controlling all others). However, students with the most initial knowledge may have benefited 
more from the TED baseline and Control conditions. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

The results of this study are consistent with the hypothesis that understanding the rationale for 
controlling variables in CVS increases the likelihood that students will remember and apply it in 
novel domains. And the benefit of emphasizing this rationale seems to be greatest for students 
who initially had the least knowledge of CVS (i.e., did not demonstrate understanding of any 
aspects of CVS). Thus, the benefit of conceptual coherence may be greater when students do not 
have anchoring knowledge in long-term memory. However, students with more initial 
knowledge may benefit more from the baseline fonn of instruction. It is possible that the baseline 
and CPO control instruction requires more effort to understand, leading to better retention (cf. 
Slamenka & Graf, 1978). 

However, due to the small sample size used in this study, these results need to be 
replicated. Planning of a replication study is underway (and should be completed in time for the 
conference). 
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Appendix B. Tables and Figures 

Not included in page count. 

Table 1. Summary of procedure. 





Computerized “TED” lesson 
(two versions: “Baseline” and 
“Added-questions”) 


“CPO” control lesson 




Introduction to lesson 




Story pretest (computerized) 


Day 1 


Part 1 of TED lesson (ramps intro, ramps 
pretest, remedial, evaluate Exp A) 


Review textbook (identify carts variables, 
review of “controlled experiment,” etc.) 


Day 2 


Part 2 of TED lesson (evaluate Exp’s B & 
C, ramps posttest) 


CPO ramps lesson 

(Identify confounds; discuss controlling 
variables; re-run experiment) 


Day 3 


Story posttest (computerized) 


3 weeks 
later 


Delayed Story test (paper) 
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Delayed posttest 



Figure 1. Condition by Story pretest interaction. 
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